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DIFFERENTIAL  ENTROPY  AND  THE  STATISTICS  OF 
INSTANTANEOUS  FAILURE 


I.  Introduction 

Instantaneous  failure  statistics  are  usually  modelled  in  terms  of  a 
probability  density  f*j(t)  of  the  (continuous)  time  to  failure,  0<t<‘»,  for  a 
component  or  system  under  consideration  [1,2].  In  order  to  define  a 
dimensionally  consistent  differential  entropy  for  such  a  process,  it  is 
necessary  to  deal  with  a  dimensionless  probability  density  that  is  a  function 
of  a  dimensionless  time  or  ordering  parameter.  This  may  be  accomplished  by 
introducing  a  positive  time  scale  parameter  so  that  the  dimensionless  time 
is  just 

t  =  t/r^  ,  0<t<«>  (1.1) 

The  corresponding  dimensionless  probability  density  is 

f^(t)  ®  Tjf^(t)  (1.2) 

Then  it  is  straightforward  to  define  a  dimensionless  differential  entropy  for 
failure  statistics  to  be 

ST  =  -  r  fT(t)  log  fT(t)  dt  *  -  E [ log  fT(t)]  (1.3) 

o 

By  explicit  demonstration  [3],  is  well-defined  for  all  the  customary  models 

a 

of  instantaneous  failure  although  S ^  is  not  necessarily  positive. 

A 

One  object  of  this  paper  Is  to  point  out  a  relationship  between  Sj  and 
the  dimensionless  relative  failure  rate  at  time  t,  i.e.  the  dimensionless 
hazard  rate,  X^(t). 

XT(t)  =  fT(t)/[l-FT(t)]  (1.4) 
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Here  Fx(t)  is  the  probability  distribution  at  time  t.  It  is  clear  that 

\x(t)dt  is  just  the  ratio  of  the  probability  of  failure  in  [t,t+dtl  to  the 

* 

probability  of  survival  to  time  t.  It  is  established  in  the  next  section  that 
is  equal  to  the  negative  of  the  expectation  of  the  logarithm  of  the  hazard 

A  A  _ 

function,  i.e.  to  -E[log  P1uS  unltv* 

This  relationship  is  then  used  as  a  basis  of  a  discussion  of  the  behavior 
ST  under  transformations  of  ordering  parameter,  which  may  occur  because  of 

A 

change  of  physical  conditions*  Transformations  that  leave  Sx  invariant  leave 

E[log  Xx(t)]  invariant*  Such  invariance  is  interpreted  as  meaning  that  the 

conditions  of  failure  remain  unchanged  even  if  measured  in  terms  of  a 

* 

different  ordering  parameter.  Similarly,  the  lack  of  invariance  of  ST  or 
E  [  log  Xx(t)]  is  interpreted  to  mean  the  conditions  of  failure  have  been 
changed  as  e.g*  in  accelerated  testing*  Both  tvpes  of  behavior  are  discussed 
further  in  section  III. 

II.  Differential  Entropy  and  the  Expectation  of  the  Logarithm  of  the 
Hazard  Rate 

As  indicated  in  the  introduction,  there  is  a  simple  relationship  between 
the  differental  entropy  §x  and  the  expectation  of  the  logarithm  of  the  hazard 
rate  E[log  A ( t )  ]  •  That  relationship  is  based  on  the  following  observations. 
First  note  that  the  cumulative  hazard  rate  Ax(t)  is  lust 

Ax(  t )  -  /t\T(t')dt'  *  logf  1-FT(  t )  ]  (2*1) 

o 

Further  note  that  the  cumulative  probability  weighted  with  the  cumulative 
hazard  rate  is  independent  of  the  choice  of  probability  density. 

fl(Fx)  =  -  A  fx(t')iog[i-FT(t')ldt' 
o 

Fx 

-  -  /  log(l-F")dF" 
o  T  T 
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( 1— FfJlogC 1— Ff)  +  Ff 


(2.2) 


A  A 

In  particular  when  F^(t)  =  1,  i.e.  as  t  +  *, 

52(1)  =  1  (2.3) 

A  A. 

independent  of  the  choice  of  f^(t). 

A  A  A  . 

It  follows  Immediately  from  the  definitions  of  S ^  and  \q»(t)  respectively 
in  (1.3)  and  (1.4)  that 

ST  =  -E  f log  XT( t ) ]  +  1  (2.4) 

A 

Hence  an  evaluation  of  S ^  for  a  particular  probability  density  serves  equally 
well  as  an  evaluation  of  Eflog  Xf(t)]«  Thus  the  differential  entropy  takes  on 
a  very  specific  empirical  meaning  In  the  context  of  the  statistics  of  instan¬ 
taneous  failures.  This  empirical  basis  is  used  in  the  next  section  Co  inter- 

A 

pret  the  invariance  properties  (or  lack  thereof)  for  under  transformations 
of  ordering  parameter. 

III.  Invariance  Properties  of  the  Differential  Entropy 

In  general  a  time  or  ordering  parameter  is  a  positive  cumulative 
function  that  increases  monotonically  from  an  origin.  A  transformation  of 
ordering  parameter  which  may  arise  from  changes  in  physical  conditions  is 
taken  to  mean  a  one-to-one  relationship  between  two  ordering  parameters  such 
that  the  derivative  of  the  transformed  parameter  with  respect  to  the  original 
parameter  is  positive  and  finite  almost  everywhere,  i.e.,  except  possibly  at 
isolated  points.  Implicit  in  the  specification  in  section  I  of  the  range  of 
the  ordering  parameter  t  to  be  0  <  t  <  *  is  the  setting  of  origin  for  failures 
at  t  •  0.  Transformed  ordering  parameters  will  here  also  be  assumed  to  be 
aligned  in  the  sense  that  they  all  share  the  same  origin. 

Consider  then  the  transformation  u  *  u(t),  u(0)  =*  0,  where  u  Involves 
the  ratio  of  a  dimensional  ordering  parameter  to  a  corresponding  time  scale 


» 
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parameter  tjj.  In  general,  such  transformations  are  not  linear.  The  one-to- 
one  nature  of  the  t rans f orma t ion  means  that 

Fu(u)  =  FT(t)  ,  u  =  fl(t)  •  (3.1) 

or 

fu(fi)  -  fT(t)/[du(t)/dt]  (3.2) 

Then 

%  -  ST  +  r  fx(t)log[du(t)/dt Idt  (3.3) 

o 

In  general  the  integral  on  the  right  hand  side  of  (3.3)  does  not  vanish  so 
that  the  differential  entropy  is  not  invariant  under  a  transformation  of 
ordering  parameter. 

In  view  of  (2.4),  (3.3)  may  be  rewritten  as 

EtlogAyCa)]  *  E[lo*XT(t)l  -  /°°fT(t)log[du(t)/dt  Idt  (3.4) 

o 

A  A 

Thus  EflogAx(t)]  is  also  not  in  general  invariant  under  the  transformation 
determined  by  (3.1).  Such  a  lack  of  invariance  is  not  unreasonable  and,  in 
fact,  should  be  expected  when  the  conditions  of  failure  are  changed.  Changes 
in  the  condition  of  failure  are  a  usual  occurrence,  for  example,  when  an 
object  being  observed  for  failure  is  subjected  to  reduced  or  increased  ambient 
temperature  in  the  course  of  a  life  test.  Consider  the  simple  case  of  a 
linear  transformation 

u ( t )  -  at  (3.5) 

where  a  is  a  positive  dimensionless  constant.  The  t ransformation  can  either 
be  viewed  as  a  change  in  the  dimensional  ordering  parameter  from  t  to  at  or  a 


change  in  the  time  scale  parameter  from  to  xy  =  x^/a.  If  a  =  1,  there  are 
compensating  changes  respectively  in  the  dimensional  ordering  ind  scale 

A 

parameters  e.g.  t  =  kt/kxTfor  arbitrary  k,  and  there  is  no  change  in  the 
differential  entropy.  If  a>l  ,  u(t)  is  larger  than  t  so  that  the  process  must 
be  slowed  to  accommodate  (3.1).  Similarly  if  a<l,  the  failure  process  must  be 
accelerated  to  accommodate  (3.1).  This  variation  of  behavior  with  the 
magnitude  of  a  is  also  revealed  by  considering  the  behavior  of  EflogX^(t)] 
under  the  transformation  (3.5).  In  that  case 

E  [  LogX|j(u)  1  =«  E[IogX^(t)  ]  -  log  a  (3.6) 

Again  it  Is  clear  that  the  failure  process  is  slowed  for  £>1  and  accelerated 
for  a<l. 

To  go  bevond  the  simple  situation  of  a  linear  transformation,  it  is 
instructive  to  consider  specific  probability  distributions  for  the  failures. 
Two  of  the  most  usual  are  the  exponential  and  the  Weibull  with  respective 
dist  ribu  t ions , 

Ft  -  l-exp-(XTt)  (3.7a) 

Fy  *  l-exp-(nyw)^  ,  8>0  (3.7b) 

Here  t  and  w  are  dimensional  ordering  parameters  and  Xj  and  py  are  dimensional 
rate  parameters.  (For  the  exponential  distribution  is  the  dimensional 
hazard  rate.)  By  introduction  of  x^  and  xy  respectively,  (3.7a,b)  are 

A  A 

readily  expressed  in  terms  of  the  dimensionless  quantities  X^*t^Xj,  t  *  t/x^, 
^y  *  Typy  anc*  w  *  w/xy.  Consider  then  the  transformation 

t  *  bw^  or  t  *  bw^  (3.8) 
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where  b  has  the  dimensions  of  [timeP'^,  b  is  dimensionless,  and  both  are 
positive.  From  (3.1)  and  (3.7a,b),  it  follows  that 

b  *  n^/X^,  b  =  (3.9) 

W  W 

The  difference  between  the  differential  entropies  calculated  respectively 
for  the  exponential  and  Weibull  distributions  is  lust 

~  =  AS-py  “  logSny/X^  -  y(8~1)/0  (3.10) 

Here  y  is  the  Euler  constant,  y  3  0.577215...  .  Clearly,  depending  on  the 

A  A  A 

values  of  3  and  the  ratio  ny/XT,  ASjy  can  be  either  positive  or  negative  cor¬ 
responding  respectively  to  decelerated  or  accelerated  failures.  The  latter 

A 

behavior  is  again  immediately  revealed  by  the  fact  that  AS^y  is  the  negative 
of  the  difference  in  the  expectation  of  the  difference  in  the  logarithms  of 
the  hazard  rate  for  the  corresponding  exponential  and  Weibull  distributions. 

A 

When  AS'py  =■  0,  the  differential  entropy  remains  invariant  under  the 

transformation  from  an  exponential  to  a  Weibull  distribution.  This  additional 

condition  serves  to  remove  some  of  the  arbitrariness  in  the  choices  of  x-y  and 

Ty.  Indeed  if  a  choice  is  made  for  one  of  the  latter  two,  the  other  becomes 

(In  general)  a  function  of  ny  and  Xq*.  Actually  it  is  convenient  to  make  the 

latter  dependence  tacit  by  choosing 
-1 

Ty  *  X^  or  *  1  (3.11) 

so  that  the  invariant  =  Sy  *  1.  Then  (3.10)  for  AS^=0  vields 

nw  -  B-1exo(  1-6-1  )y  i  A($)  (3.12) 

Here  the  right  hand  equality  serves  to  define  A(0)  so  that 

TW  -  A(0)n-1  (3.13) 

w 
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which  determines  the  dimensional  scale  parameter  in  terms  of  the  rate 


parameter  nw.  It  follows  directly  from  (3.9)  and  (3.12)  that 

b  =  [A(8)l®  “  bx ^/tt  (3.14) 

W 

or 

xT  =  b[TW/A(6)]S  (3.15) 

Thus  the  respective  dimensional  time  scales  are  related  bv  the  same  trans¬ 
formation  (3.8)  as  the  dimensional  order  parameters  provided  there  is  an 
appropriate  normalization  determined  by  the  condition  A§TW“0. 

A 

The  condition  AS^=0  may  be  understood  to  mean  that  the  physical  mecha¬ 
nisms  of  failure  are  not  changed.  For  example,  one  could  take  the  view  that 
an  object  has  its  own  internal  clock  which  orders  its  evolution  to  failure. 

The  clock  used  in  the  laboratory  for  measurements  in  general  will  be  different 
from  the  internal  clock.  Since  in  such  a  situation,  the  phvsical  mechanisms 
of  failure  remain  the  same  independent  of  the  ordering  parameter  used  for 
enumeration  of  failures,  E [ log^fC t ) ]  and  hence  also  would  remain  invariant. 
When  this  invariance  obtains  under  the  transformation  (3.8)  as  for  example 
when  the  measured  distribution  is  Weibull  with  an  underlying  exponential,  the 
measured  failure  rate  should  have  some  surprising  features.  For  instance 
if  8>1  ,  it  should  be  smeller  than  expected  and  similarly  if  B<1 ,  it  should  be 
larger  than  expected.  Another  possible  indicator  of  the  presence  of  an  under¬ 
lying  distribution  here  would  be  unexpected  dependencies  of  the  rate  on  system 
specific  parameters.  For  this  example,  if  the  change  of  a  power  of  parameter 
to  a  (power)/0  provided  expected  behavior,  then  one  would  have  an  indicator 
that  there  is  an  underlying  exponential  distribution*  This  latter  indicator 
also  provides  the  key  to  improved  performance  by  its  revelation  respectively 
of  an  increased  or  decreased  (depending  on  the  value  of  6)  sensitivitv  of 
parameters.  Appropriate  modification  of  the  parameters  entering  in  the  rate 
parameter  itself,  could  lead  to  improved  performance. 


7 


The  example  given  here  exhibited  a  relationship  between  the  exponential 


distribution  into  the  Weibull  distribution  bv  means  of  a  transformation,  Eq. 
(3.8).  This  discussion  suggests  a  useful  general  method  of  finding 
relationships  between  different  distribution  functions  and  their  respective 
entropies.  Two  other  examples  of  interest  in  Statistics  of  Instantaneous 
Failures  would  be  (a)  g-Weibull  to  g^-Weibull,  and  (b)  exponential  to 

bathtub  distribution. 

The  case  of  transformation  of  a  Weibull  to  another  Weibull  is  of  some 

interest.  In  the  notation  of  (3.10)  it  is  found  that 

SW  ~  =  ASy^  =*  log(S'rr/8ny)  +  ($-&'/$$'  )y  . 

w 

The  discussion  of  the  type  following  (3.6)  in  this  case  shows  that  the 
failure  process  is  slowed  or  accelerated  depending  on  the  relative  values  of 
the  rates  ny  and  riy^  as  well  as  the  exponents  6  and  S'  • 

The  discussion  above  concerned  transformation  which  carried  the  range  of 
the  ordering  parameter  variable  from  (0,«)  to  (0,®).  It  is  also  possible  to 
make  other  types  of  transformations  which  carry  (0,®)  to  (a,b)  or  to  (-<®,®). 

In  a  generalized  sense,  these  may  also  be  considered  as  transformations  of 
time  but  the  physical  meaning  is  perhaps  obscured  by  such  trans f ormat ions. 
Thus,  we  could  use  the  transformation  Xx  «  log(l+e^v)  (b>0)  which  carries 
x:(0,®)  to  y:(-®,®).  and  the  exponential  distribution  in  x  is  then  trans- 

C.  *  ^  *  a 

formed  to  a  logistic  distribution  e~Dy / ( l+e~by )z .  The  entropy  change  Sx-Sy 
is  (-1-Jfcn( X /b) ) •  The  transformation  Xx  *  log( b-a/b-z ) ,  a<z<b,  x:(0,®)  becomes 
z:(a,b)  and  the  exponential  distribution  becomes  a  uniform  distribution  in  z. 
The  change  in  entropy,  Sx-S^  1 -£n [ X (b-a) ] . 

The  bathtub  distribution  is  obtained  from  an  exponential  distribution 
when  the  transformation 

At  =»  e(yz)  -1,  b  >  0 


» 
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is  made  on  the  exponential  distribution  in  the  t -variable.  This  takes  t:(0,*>) 
to  z:(0,»)  and  the  exponential  distribution  is  transformed  to  the  bathtub 
distribution  in  the  z-variable.  The  entropv  difference  is  given  bv 

S^-S  -  -  £n(Tb  )  -/°°e~v£n(  L+v)dv  -  (k~l  )  f°°  e~^dv£ n(  £n(  1+v )  ) 

“  1  T  0  b  0 

In  summarv,  we  have  related  the  dimensionless  entropy  for  failure 
statistics  to  the  negative  of  the  expectation  of  the  logarithm  of  the  hazard 
rate.  We  have  also  pointed  out  that  the  general  lack  of  invariance  of  the 
differential  entropv  under  a  transformation  may  be  viewed  as  equivalent  to 
changing  the  physical  conditions  under  which  the  hazard  function  is  measured. 
In  future  work,  we  plan  to  use  this  general  observation  as  a  basis  for  the 
development  of  strategies  for  accelerated  testing. 
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